naybe reuse available_ here ?

We can, I added this attribute for tracability, as if someone calls getInstance after cleaning the backend, it's for sure a lifecycle bug and I thought it deserves its own error message to better trace it, instead of just getting "Backend not available" without knowing that it comes from a code mistake.

But your call, let me know what you prefer.

You could still use available_ here to print the same message. I personally think it is harder to reason about several dependent internal states. The question is: "in what situation exactly is it useful to have both available_ and cleaned_up_ state?". I fail to see such a case.

But it's really your call at the end!

Btw, cf. other comment, the lifecycle should imo be managed by Communicator.

btw, we could even argue that available_ is redundant with impl_ == nullptr

There is a case where impl is not null but just points to a trivial impl
But the right solution here would be just not to instanciate impl if it cannot be instanciated

is that really necessary? It looks suspicious to me, can you help me understand?

It's because UCX build can lack CUDA support, and registerMem silently succeeds but misclassifies VRAM as host memory behind the scenes, and the only way to check this is to actually call prepXferDlist on an example tensor (which fails in the case ucx doesn't have cuda).
Without this probe, NIXL would be marked as available and registerMem would work, but then fail much later on prepareTransfer or postTransfer. As of my experience, it can be pretty confusing.

Ok I see, so let's keep this check for now. Is UCX/NIXL team aware of this bug and plan to fix?

I opened a PR in nixl: ai-dynamo/nixl#1393

This wait function is cpu blocking so in practice it is more or less unusable in our context. Do you have an idea how to make this not blocking for cpu -- and ideally cuda-graph capturable?

busy-wait spin loop will consume CPU while waiting for transfer completion

consider adding std::this_thread::yield() or a small sleep inside the loop to reduce CPU usage

Compile error when USE_NIXL is not defined

All seven NixlBackend public API methods (lines 432–471) unconditionally call through to impl_->registerTensors(...), impl_->deregisterTensors(...), impl_->exchangeMetadata(), impl_->prepareTransfer(...), impl_->postTransfer(...), impl_->getTransferStatus(...), and impl_->waitTransfer(...).

When USE_NIXL is not defined, NixlBackend::Impl is the empty stub class NixlBackend::Impl {}; (line 403). None of those methods exist on the empty class, so every delegating call is a hard compile error:

error: 'class nvfuser::NixlBackend::Impl' has no member named 'registerTensors'

Because nixl.cpp is added to NVFUSER_SRCS unconditionally (CMakeLists.txt line 251), any build with NVFUSER_STANDALONE_BUILD_WITH_NIXL=OFF (the default) will fail to compile.

The fix is to either guard these wrapper methods with #ifdef USE_NIXL, or add stub declarations to the #else branch of the Impl definition so that the calls are still valid (and simply unreachable at runtime due to the NVF_CHECK(isAvailable(), ...) guard):

#else // !USE_NIXL
class NixlBackend::Impl {
 public:
  void registerTensors(const std::vector<at::Tensor>&) {}
  void deregisterTensors(const std::vector<at::Tensor>&) {}
  void exchangeMetadata() {}
  NixlTransferHandle prepareTransfer(
      const std::vector<TensorDesc>&,
      const std::vector<TensorDesc>&,
      int64_t,
      NixlXferOp) {
    return {};
  }
  void postTransfer(NixlTransferHandle&) {}
  NixlXferStatus getTransferStatus(const NixlTransferHandle&) const {
    return NixlXferStatus::kError;
  }
  void waitTransfer(NixlTransferHandle&) {}
};
#endif // USE_NIXL

Deadlock if loadRemoteMD fails on any rank

If agent_->loadRemoteMD(...) returns a non-success status, NVF_ERROR at line 284 throws an exception on the failing rank. That rank never reaches the communicator_.barrier() at line 293. All other ranks, however, proceed to call barrier() and block indefinitely, producing a distributed deadlock.

The barrier here is intended to prevent premature key deletion, but the error path bypasses it. Consider either:

1. Catching the error, reaching the barrier (to unblock peers), and re-throwing afterward, or
2. Using a try/catch or a status variable to ensure barrier() is always called before propagating errors:

nixl_status_t failed_status = NIXL_SUCCESS;
int64_t failed_rank = -1;
for (int64_t rank = 0; rank < world_size; ++rank) {
    if (rank == my_rank) continue;
    auto bytes = store->get(md_key_prefix + std::to_string(rank));
    nixl_blob_t remote_md(bytes.begin(), bytes.end());
    std::string remote_agent_name;
    nixl_status_t status = agent_->loadRemoteMD(remote_md, remote_agent_name);
    if (status != NIXL_SUCCESS && failed_status == NIXL_SUCCESS) {
        failed_status = status;
        failed_rank = rank;
    }
}
// Always reach the barrier so other ranks are not left hanging.
communicator_.barrier();
store->deleteKey(md_key_prefix + std::to_string(my_rank));
NVF_ERROR(
    failed_status == NIXL_SUCCESS,
    "NIXL loadRemoteMD failed for rank ", failed_rank,
    " with status ", static_cast<int>(failed_status));
metadata_exchanged_ = true;

Missing indentation on closing brace

The closing brace of getTcpStore() at line 129 has lost its 2-space class-body indentation, which is inconsistent with every other method in this class.

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}